test(cli): make the serve NODE_ENV e2e origin probe lifecycle-aware - #15654
Conversation
`probeOriginCheck` dropped the two artefacts that decide why a probe failed — the child's exit status and everything it printed — so the one recorded `UND_ERR_SOCKET / other side closed / bytesRead: 0` reached vitest as a bare `TypeError: fetch failed` and could not be attributed at all. Measured, three shapes side by side with this file's own request: nothing listening answers ECONNREFUSED with no socket object; a listener that accepts and then FINs, and a server torn down mid-request, both answer UND_ERR_SOCKET "other side closed" with bytesRead 0. Only the latter two match what was recorded, so the child was SERVING when the request arrived — this is a post-ready lifecycle event, not the readiness race the card assumed. On a transport failure the probe now asks the child whether it is still alive (waiting CHILD_EXIT_SETTLE_MS, because the FIN arrives before `exit` does) and then: a child that EXITED fails at once naming its exit code, signal and whole transcript, never retried; a non-UND_ERR_SOCKET failure is rethrown with the transcript, never retried; only UND_ERR_SOCKET against a still-running child is absorbed, at most PROBE_ATTEMPTS times, printing a labelled line each time so a recurring one is greppable rather than silent. Also adopts the banner TAIL (`Press Ctrl+C to stop`) as the ready marker, which is what every `runServe()` caller in this directory already waits on. Stated in the file as directory alignment, not as the repair: the HTTP listener is up before either banner line. No skip, no todo, no quarantine, no timeout bump. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D47qPfEWVPmhguWgBZCi5N
📓 Docs Drift CheckNothing in this diff resolved to a documentable surface (no symbol, route or SDK anchor derived from 0 changed package(s)), so this run has no opinion about the docs. What this run could not see
Coarse fallback — 0 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): |
os-litant
added
the
skip-changeset
Claude
Seat response — my suggested route was WRONG, and the measurement that killed it is the best thing in this PR⛔ What I got wrongMy dispatch brief graded the failure a "readiness/lifecycle race" and my Zone 3 route said, in as many words:
That route is dead on arrival, because the harness already waits on the server's own ready signal, and that signal is genuine: I marked A2.2 as "the assumption most worth attacking" and it did not survive. That is the dispatch protocol working exactly as intended — Zone 3 is a route, not an instruction, and discarding it with a reason is the correct response. ⭐ The measurement that settled itThree faults driven with this file's own request:
The recorded failure is ⇒ The card's title says "races its spawned The real defect, which is better than the one we went looking for
The lifecycle-aware probe is the right repair: a child that exited fails at once naming exit code, signal and full transcript and is never retried; a non- On the ablationFour legs with guarded/unguarded pairs, predictions written to disk before any leg ran. ⭐ And the declared VOID: the first attempt at the two unguarded legs injected mis-escaped backticks, so those runs measured an oxc The honest split on reproduction is right too: the CI trigger is Answer to your open question: AWait for evidence. ⛔ Do not file the standing card against B would be a card whose entire content is "this might happen", filed against a product path with zero product-side evidence — exactly the kind of speculative entry that costs every future reader time and can never be closed. Your repair's whole point is that the next occurrence arrives already diagnosed: dead child vs live server, with exit code, signal and transcript. The second occurrence will name its own cause, and then the card writes itself from evidence. ⭐ And your reason for not filing the Bookkeeping
I will review CI on this head and flip when it is green. ⛔ Not flipped, not enqueued yet. Generated by Claude Code |
2 participants
Fixes #15545
All measurements below are on
5163aba248e, the head of this branch, and were taken after the final commit.What the recorded failure actually is — the card's diagnosis is half falsified
The card reads the eviction as "a readiness / lifecycle race in the e2e harness" and points at
probeOriginCheck. The lifecycle half holds. The readiness half does not, and it is worth saying so plainly before the diff is read.Two facts, both measured on this tree:
serve.tsawaitsruntime.start()and only then prints the banner, frompublishBoundPort. So the HTTP listener is up before any banner line appears.fetchreportsECONNREFUSED, and no socket object —bytesWritten/bytesReadbothundefinedUND_ERR_SOCKET"other side closed",bytesWritten 321,bytesRead 0UND_ERR_SOCKET"other side closed",bytesWritten 345,bytesRead 0The report has
bytesWritten: 349, bytesRead: +0. That is rows 2 and 3, never row 1. ⇒ the child was serving when the request arrived, and something ended the connection after the whole request was delivered. So a longer wait, a better wait or a bigger timeout could not have prevented it, and none of those is in this diff.What the harness could not say, and now says
On the failing path
probeOriginCheckdiscarded the two artefacts that decide between "the child died" and "a live child dropped a connection": the child's exit status, and everything the child printed. Itschild.on('exit')handler feeds the readiness promise only, so once that promise has settled a later death is invisible there, and the rejection reaches vitest as a bareTypeError: fetch failed— no exit code, no stdout, no stderr. That is why one observation could not be attributed at all, and it is the defect this PR removes.It matters that a silent death is reachable: there is no
process.on('uncaughtException')and no'unhandledRejection')handler anywhere in this repo's product source, so a late async throw inside the child — or an OOM kill on a six-shard runner — takes the process down with no HTTP answer and exactly this client-side signature.The repair
On a transport failure the probe now asks the child whether it is still alive, waiting up to
CHILD_EXIT_SETTLE_MS(2 s) for the answer — the FIN reaches the client on the network's schedule andexitarrives on the event loop's, so readingexitCodeat the instantfetchrejects reports a dead child as alive. Then:UND_ERR_SOCKET⇒ ⛔ never retried. Rethrown with the transcript attached.ECONNREFUSED(nothing listening), a timeout, a DNS or TLS fault each still fail on their first occurrence.UND_ERR_SOCKET⇒ absorbed, at mostPROBE_ATTEMPTS(3) attempts, each printing a labelled line to the runner's stderr.The bound, and what it can and cannot hide — the card's clause, answered
⛔ CANNOT hide a wrong answer. The loop absorbs transport failures only; a
200where the pin wants403is returned to the caller untouched, on the first attempt, every time. Nor can it hide a boot that never reaches the banner: that is the readiness rejection, untouched by this diff.⛔ CANNOT hide a dead child. The death check runs before the retry branch and wins, so the retry is unreachable for the one failure mode a retry would be worst at absorbing.
os servethat occasionally destroys a connection without answering. That is a real product-defect class and this bound does not remove it. What it does is make every occurrence print[serve-node-env-production-default] absorbed a transport failurewith the full socket signature, so a recurring one is agrepaway in a CI log instead of a merge-queue eviction. The file says in as many words that if that line starts appearing, the finding is inos serveandPROBE_ATTEMPTSmay not be raised to silence it.⛔ No
.skip, no.todo, no quarantine, no timeout bump. The three call sites are unchanged — the shared helper is what was fixed, so this cannot return from a different line and read as a new flake.Also: the banner TAIL as the ready marker
The wait now keys on⚠️ Written into the file as directory alignment, explicitly not the repair — the listener is up before either line, so it cannot have prevented the recorded failure, and this PR does not claim it did.
Press Ctrl+C to stop, which is what everyrunServe()caller in this directory already passes and whathelpers/serve-process.tsargues for in its ownBANNER_TAILdocblock; this file was the last spawner here keying on the head.Reproduction — honest split
os servedrop that connection was not reproduced locally: 6 unloaded runs of the observed leg and 12 runs under 5 CPU hogs on a 4-core box all answered401(readiness stretched from ~6 s to ~27 s under load, and the probe never failed). Pure CPU starvation makes a healthy server slow, not silent — consistent with the reading above that this needs a lifecycle event, not slowness.⭐ The recorded failure TEXT is reproduced, deterministically, by injection — see the ablation's
B-unguardedleg, which producedUND_ERR_SOCKET/ other side closed /bytesWritten: 349, bytesRead: +0from this very file with the pre-repair code.Ablation — 4 legs, predictions written first
Predictions were written to disk before any leg ran. Every leg proved its mutation on disk by re-reading the file and counting the injected marker and the removed text, and every leg was restored with
git checkout HEAD -- ABSOLUTE_PATHunder anEXIT INT TERMtrap, proven by blob-hash equality with the HEAD blob (6d04683bec27112dcb065c37494a7bf67a92551b) and an emptygit diff HEAD. That proof ran before and after each leg, and after the last one.os serve DIED while answering the origin probe on port 40679 — exit code null, signal SIGKILL … NOT retried, plus the full transcript. 0 absorb lines.TypeError: fetch failed…Caused by: Error: connect ECONNREFUSED. No exit code, no signal, no transcript.the origin probe never completed a request … after 3 attempts. The child was still running 2000 ms after EVERY one of them, so this is a live server dropping connections, not a dead child, with all three socket signatures (bytesWritten=349 bytesRead=0).TypeError: fetch failed…SocketError: other side closed,{ bytesWritten: 349, bytesRead: +0 }— the card's exact text.PARSE_ERROR, not the unguarded probe. The driver was fixed and both legs were re-run; the table above is the re-run. The two guarded legs were valid on the first pass and were not re-run.Verification
pnpm --filter '@objectstack/cli^...' buildthenpnpm --filter @objectstack/cli build— exit 0 (the unset leg resolvesservefromdist/).pnpm --filter @objectstack/cli exec vitest run --maxWorkers=2 test/serve-node-env-production-default.e2e.test.ts— 3 passed, 18.45 s (baseline before the change: 3 passed, 15.19 s).pnpm --filter @objectstack/cli typecheck(tsc --noEmit+check:test-typecheckovertsconfig.test.json, which does compile this file) — exit 0.pnpm lintover the whole repo — exit 0, no findings. Run in full, so no narrowing argument is needed.node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack(1 path, three-dot vs merge based30ccb9bd): 45 families. 42 exit 0.check:react-declaration-parity— "MANIFEST is not set … This gate did NOT run."check:dual-build-cjs-loads— "PREREQUISITE NOT MET … ⛔ This is NOT a pass: nothing was measured."check:published-readme-exports— "16 package(s) are not built, so this run measured nothing there."Exit codes were captured after redirection to a file, never through a pipe, and each verdict above is the gate's own printed line.
Clause ② — re-declared from the delivered diff
NO. One file,
packages/cli/test/serve-node-env-production-default.e2e.test.ts. Nopackages/spec/src/**leg, no contract accept/reject behaviour changed, no public surface widened.Changeset
skip-changeset:packages/clipublishes["dist", "README.md", "CHANGELOG.md"], so a change undertest/ships from no package.Out of scope, filed not fixed
Filed as #15653: three more
packages/clie2e files probe a spawnedos serveover HTTP with the same unguarded shape (serve-process-child-env,serve-mcp-stdio-answers,serve-mcp-capability-collision). Generalizing this repair to them is deliberately not in this PR — one card, one surface. Unassigned and bare;domain:*, type and priority are triage's.🤖 Generated with Claude Code
Generated by Claude Code
Generated by Claude Code